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The Internet is considered to be one of the most important mechanisms for sharing research 
(Chavkin & Chavkin, 2008), raising questions about the scope and variety of research-based 
education resources available online. Whereas years ago, the challenge for educators was to 
find relevant research information for practice, this challenge has shifted to one of sorting 
through the “infoglut” (Edmunds & Morris, 2000), or the abundance of information claiming 
to be grounded in research, and being able to judge which resources are most valuable and 
reliable. Drawing on scholarship that seeks to mobilize evidence between researchers, 
practitioners, and policymakers (Levin, 2011), this study reports the results of a systematic 
search of online research-based resources for educators across seven subject areas. Our 
descriptive typology categorizes 333 products along various dimensions (e.g., producer 
organization, target audience, country of origin) and demonstrates a surprising variety 
claiming to be research-informed. 
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Introduction 

Just a couple of decades ago, educators would have had difficulty finding research-based resources 
connected with their practice (Levin, 2010). Unless they ran across materials through happenstance, 
they would have had to consult paper or microfiche archives, and write letters asking for copies of 
various resources to be sent to them. That landscape has changed dramatically with the advent of 
the Internet as an important medium of research dissemination (Chavkin & Chavkin, 2008; 

Waddell, 2002) with its online databases, social networking tools, and organizational Web sites 
(Arduengo, 2008). Indeed, recent advancements in technology have opened floodgates of information 
(Feather, 1998), as well as opportunities by which it can be accessed and shared. A simple search on 
the Internet can generate millions of results. As a deluge of information is at our fingertips, we are 
now left with the exhausting and overwhelming task of sorting through this “infoglut” (Edmunds & 
Morris, 2000, p. 18) to find quality information that is relevant and also reliable (Brawden & 
Robinson, 2000). One would need to develop the habits and knowledge to filter through what and 
how much one should consume (Johnson, 2011). 


Please address queries to: Nathalie Carrier, University of Toronto. Email: nat.carrier@utoronto.ca 





Carrier, Sohn, Jao, & Shah, 2013 


In the field of education, there is a growing interest in how to share and use research to improve 
policy and practice in schools and school systems (Cooper, Levin, & Campbell, 2009). Although the 
Internet is now a main way to do this, few studies have explored the use of technology systems as 
facilitators of these efforts (Dede, 2000; Edelstein, Shah, & Levin, 2011). Our team (consisting of five 
doctoral students and one faculty member at a school of education) could find no other studies that 
address the range of resources educators are likely to encounter on the World Wide Web, nor how 
they might go about swiftly and effectively navigating them. 

This study reports the results of a systematic online search of resources available to educators. The 
intent of this search was twofold: (a) to explore systematically the online environment facing 
educators when searching for research-based resources and (b) to understand the ways producers 
share their research-based resources with educators through the Internet. When searching for the 
resources, we used “research/evidence” or “research-based/evidence-based” in the search string as a 
proxy for identifying those resources claiming to be informed by or that drew from research. 

However, it is important to note that the resources we identified may or may not in fact be research 
based. Our study asks more generally, “What kinds of research-based resources are online?” We 
organize our search results of 333 selected resources from seven subject areas along several 
dimensions, including target audience, affiliated organization, and country of origin. 

Information Overload in Education 

The ability to access information has been steadily on the rise since the advent of printing. Studies 
have shown that an abundance of information can paralyze rather than enable action and can reduce 
one’s sense of control (Edmunds & Morris, 2000). Often referred as “information overload” (e.g., 
Bawden, Holtham, & Courtney, 1999; Eppler & Mengis, 2004; Shenk, 1997; Sullivan, 2004; Wurman, 
2001), the amount of information available may be so overwhelming that consumers may no longer 
be able to process, organize, and use it effectively (Feather, 1998). 

To exacerbate the problem, the Internet is not selective in terms of who can post information, nor are 
there checks and balances to ensure that all information on the Internet is accurate. In such an open 
environment the complications are many. First, the nature of the search process may affect what an 
educator finds. Once information that seems to be aligned with the search goals has been found, 
there is the added difficulty of discerning whether what was found is reliable. As Bawden and 
Robinson (2009) state, “Even when the [sponsoring] originator is known without doubt, queries of the 
veracity and reliability of what is presented [exist]” (p. 182). And even assuming that the 
information presented is credible and accurate, one would then need to determine which items from 
the search process best meets the needs of a user. It can be very challenging to sort through large 
quantities of information to “locate the small piece that we need” (Nelson, 1994, p. 13). In a 
profession where time is often at a premium, teachers cannot afford to feel dragged down by the 
“data smog” (Shenk, 1997). 


Theoretical Perspective: Knowledge Mobilization 

To help understand the context of information overload teachers may experience in their online 
searches, we draw from an area of work called knowledge mobilization (KMb), or efforts to connect 
research more strongly with policy and practice. KMb has become a salient characteristic of policy 
and practice reform agendas worldwide. The area has garnered considerable interest among public 
policymakers, researchers and practitioners; likewise, KMb research has been conducted in a range 
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of fields such as public policy (Landry, Amara, & Lamary, 2001), management (Hemsley-Brown, 
2004), and education (Levin, 2011; Sebba, 2004; Walter, Nutley, & Davies, 2005). 

We draw specifically from Levin’s (2011) model, which identifies three intersecting domains in which 
KMb takes place in education: (a) “producers” of research—typically in universities, but also in 
nonprofit organizations, governments, and school systems themselves; (b) “users” of research, such 
as government departments of education, teachers and school leaders, school districts, and 
professional organizations; and (c) intermediaries who disseminate research for various purposes, 
such as the media, interest groups, professional development providers, and commercial or nonprofit 
organizations. Any given organization can be working in all three domains. The value and impact of 
research is contingent upon activities in all three domains. These processes are complex and 
multidimensional: the impact of research can be indirect and occur over a long period of time (Weiss, 
1979), and is influenced by broader social forces such as the political climate, demographics, popular 
culture and media interests (Hargreaves, 1999). KMb themes that have been explored across 
disciplines include the following: 

• Characteristics of the research or its communication (relevance, clarity, timeliness, 
availability, communication mechanisms; e.g., Cordingley, 2008; Lavis et al., 2005; Figgis, 
Zubrick, Butorac, & Alderson, 2000) 

• Capacity of the professional community to find and use the research (resources, skills, 
attitudes to research, priority in the workplace; e.g., Coburn & Talbert, 2006; William & 
Coles, 2003; Zeuli, 1994) 

• Relationships between researchers and those with a potential interest in the work (direct 
connections, role of third parties; e.g., Cooper, Rodway, & Read, 2011; Corcoran, Fuhrman, & 
Belcher, 2001). 

There are opportunities to improve KMb in each of these areas, and many such efforts have been 
described, including communicating research results in different ways (multiple publication forms, 
electronic communication, Web sites, tailored summaries), building links with potential users 
(networks, seminars, advisory groups, participation in professional conferences), and purposeful 
work by third parties (e.g., cultivating media relationships; Walter, Nutley, & Davies, 2003, 2005, 
2007). 

Using Research: What Counts? 

In the field of education, fads and bandwagon approaches are often adopted even without supporting 
research (Cuban, 1990; Slavin, 1989). As Heward (2003) describes, “Convention, convenience, dogma, 
folklore, fashion, and fad—more so than the results of scientific research—have all influenced theory 
and practice in education over the years” (p. 199). In addition, though much is already known about 
effective educational practice (Hattie, 2009), this knowledge fails to shape many ventures and reform 
efforts that spread, and which ultimately lead to unsuccessful and wasteful innovations (Levin, 

2010). This suggests that certain practices catch on not because of their grounding in research or 
evidence of efficacy, but due to other factors. This situation also raises important questions around 
the barriers associated with accessing and using research in practice. 

Although many education professionals are interested in education research (Berhstock, Drill & 
Miller, 2009; Biddle & Saha 2002; Cordingley, 2008; Landrum, Cooke, Tankersley, & Fitzgerald, 


Journal of Educational Research and Practice 


35 



Carrier, Sohn, Jao, & Shah, 2013 


2002), certain factors can inhibit them from using it in their practice. For instance, educators may 
hold negative attitudes towards research, sometimes finding it irrelevant, contradictory and 
inaccurate (Fleming, 1988, as cited in Malouf & Schiller, 1995). Many teachers lack training in the 
principles of research design, and might consequently develop false beliefs after “reading research 
that in fact [it] does not address the issue of whether a practice works” (Cook, Tankersley, Cook, & 
Landrum, 2008, p. 77). 

Important to consider, are the potential biases towards interpretations of research (Coburn & 
Talbert, 2006; Spillane, 1998). Teachers, as Fuhrman (1992) puts it, “translate research findings 
through the lens of prior knowledge and understanding, thus making sense of new knowledge in the 
context of their daily activities” (p. 8). The influential nature of prior belief poses challenges to 
changing practice. According to Kennedy (1997), teachers who are sympathetic to reform may be 
more receptive to research findings, but those who are not open to change on a given issue are 
unlikely to be persuaded by research findings, regardless if the research directly informs their work. 
This is consistent with the fact that people in general are less inclined to pay attention to evidence in 
areas where they have deeply held beliefs (Pajares, 1992). 

Numerous studies demonstrate the significance of the perception of relevance over evidence (Borg, 
2009; Honig & Coburn, 2008; Joram, 2007; Levin, 2008 Zeuli, 1994). In Cordingley’s (2008) words, 
“practitioners need to connect intellectually, practically and emotionally with the knowledge they are 
offered in the research accounts if they are to take it on board and use it in their practice” (p. 37). 

In addition to perceived relevance, utilization of research increase when research based products are 
customized to specific audiences, particularly because research is often produced in formats not 
designed for use by policymakers or practitioners (Cooper et al., 2011). Customizing research 
products to practitioner groups could lead to increased utilization (Boardman, Arguelles, Sharon, 
Hughes, & Janette, 2005; Cordingley, 2008; Figgis, Zubrick, Buorac, & Alderson, 2000; Lavis et al., 
2005). 

In order to make research more accessible to teachers, and ultimately increase user uptake, 
Cordingley (2008) suggests research be presented in a more accessible form, with the use of shorter 
formats, summaries of the methods and user-friendly language that is clear, simple and free of 
jargon. These recommendations are in line with Williams and Coles’ (2003) study, which—not 
surprisingly—showed that teachers preferred immediately accessible sources of information, as they 
lacked the confidence to define a search strategy to seek out research based materials themselves. 
The teachers were overwhelmed when faced with a wide range of choices for information, and were 
not comfortable assessing the quality of research in terms of appropriate methodology, sufficient 
evidence, and other criteria required for high quality research. In addition, when teachers were 
invited to rank a range of potential barriers to the use of research evidence, lack of time was the 
most frequently cited challenge (William & Coles, 2003). Furthermore this study showed that 
regardless of time constraints, the teachers found the process of seeking and evaluating research 
information to be a challenge. They were uncertain about how to employ search strategies, and even 
less confident in the area of evaluating and using research. Many of the judgments they made about 
research appeared to be less concerned with quality in terms of validity and reliability of evidence 
and more concerned with whether the research addressed classroom reality. They therefore tended 
to allocate greater value to factors such as applicability of the research topic and conclusions than to 
other criteria such as appropriate methodology, lack of bias, and sufficient evidence to support 
conclusions. Several studies build on these points by exploring the specific preferences educators 
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may have around format of presentation of the research (Blamires, Field, & Wilson, 2010; 
Cordingley, Bell, Evans, & Holdich, 2005; Centre for the Use of Research and Evidence in Education 
[CUREE], 2007; Cordingley & The National Teacher Research Panel, 2000). 

The preponderance of empirical work in KMb has focused on the domain of research production; 
there is a need for more research on educators’ perspectives particularly as it has been shown that 
research is deeply affected by social processes within organizations (Coburn, Honig, & Stein, 2009; 
Mitton, Adair, McKenzie, Patten, & Perry, 2007). In our review, we could find no studies exploring 
the scope of what is available online; however, Edelstein, Shah, and Levin’s (2011) study tracking 
the use of several education organizations’ Web sites via Google analytics software demonstrated 
that few visitors to the sites accessed and retrieved research based resources posted by the 
organizations, suggesting that improvements can be made in how producers share this information. 

Method 


Systematic Online Search 

We began this project with an interest in understanding the array of research-based resources 
available to educators online. We hoped to get a sense of what was “out there”—what resources were 
available for educators, and from what sources. We were interested in seeing if patterns could be 
found between research-producing organizations and the kinds of resources available, by subject 
area, by type of resource, by country, by sector, and by type of organization. 

With these questions in mind, we embarked on a systematic online search of research-based 
resources for educators. It was also understood that we would not be able to understand how 
educators access and make sense of “research-based resources” without having something tangible 
they could examine. We therefore decided to search for educational resources that educators could 
implement in their practice (as opposed to other kinds of resources such as professional networks or 
events). These came in a wide variety including tipsheets, handbooks, curricula, guidelines, software 
programs, blogs, videos, encyclopedia entries, and many others. 

Because the field of education covers so much ground, we chose to search for these resources in seven 
specific subject areas with separate searches for each area in order to get a sense of the kinds of 
resources available to educators across a variety of topics. The seven areas we selected were intended 
to cover diverse areas of education and included mathematics, leadership, second language 
education, arts, behavior management, literacy, and special education. 

We reviewed information describing effective Google search strategies (for example of how to 
combine and simplify search terms, see 

http://support.google.com/websearch/bin/answer.py?hl=en&answer=134479) and created a list of 
search terms for product type, provider, sector, and topic that were used in our search strings. We 
used “research/evidence” or “research-based/evidence-based” in our search strings as a proxy for 
searching for products that claimed to be informed by or drew from research, regardless of quality or 
quantity. It is important to note that these resources may or may not have been grounded in 
research. Our search strings also included the subject area of interest plus “resource” or “product” or 
a specific type of product. A few examples of such search strings included Teacher 
Resources+Literacy+Evidence Based; education leadership+toolkit+evidence; 

mathematics+resource+evidence; and learning disability research products for educators. All of the 


Journal of Educational Research and Practice 


37 



Carrier, Sohn, Jao, & Shah, 2013 


searches were done through Google. The team met several times to discuss construction of search 
strings so that we would have comparable results. 

Our search process efforts were in part inspired by a technical report published by CUREE in 2007. 
The CUREE (2007) researchers conducted a systematic review of research studies on the topic of how 
practitioners “engage in and/or with research to inform and develop their practice” (p. 5), utilizing 
online database search engines. In the CUREE report, the review of studies is divided into several 
themes of interest (range of approaches to practitioner engagement, application of research, 
differences in engagement, etc.) and organized into a descriptive map synthesizing different strands 
of studies. Similarly, we sought to organize our online search results into a descriptive typology with 
categories that included subject area, country of origin, target audience and affiliated organization. 
Our search process consisted of the following specific steps: 

1. A typical search produced huge numbers—often millions—of hits, so had many thousands of 
pages of results. For each search string, we randomly selected two pages of hits out of the 
first 10 pages. We captured and saved these pages as screenshots. We selected from the top 
10 pages to find those resources Google deemed to be most relevant to the string at hand. 

2. From the two chosen pages, we selected as many products as seemed, based on a quick 
inspection, to meet our criteria for being research-based and relevant to educators. If the 
product was a direct link, we included it; if it was a link to an organization, we searched the 
homepage of the organization for product examples. 

3. We continued this process until we had used 15 different searches or reached a total of 50 
products on this topic. 

4. Inevitably, deciding which products or Web site links were relevant to our study involved 
some degree of judgment, though many links were simply not relevant at all in several 
respects, as would be expected from a search—for example, if the link was not a product, was 
not related to the subject area of interest, etc. 

In this search process, we kept track of both the number of hits per search string, as well as the page 
numbers of those two pages we selected at random. Six out of seven of the subject areas reached 50 
products before the 15 search strings (the exception was the arts at 33 products); therefore, our 
Microsoft Excel listing of products totaled 333. These were then merged into one Excel document. 

We do acknowledge several limitations in our search approach. Some searches used more specific 
search strings (e.g., a specific type of product in the search string as opposed to the more general 
“product” or “resources”); some team members selected only resources that were a direct hit on the 
search page while others linked to an organization’s Web page and selected resources offered by that 
particular organization, and some team members included resources that cost a fee while others 
excluded fee-based resources altogether. While consistency in the search process could have been 
improved, we do believe our data provide interesting and consequential findings about the research- 
based resource environment available to educators. 

Selection of Cases and Descriptive Typology 

The resources we found varied widely in format and included e-modules, tip sheets, lesson plans, 
software, curriculum guides, toolkits, handbooks, strategies, conceptual frameworks, meta-analyses, 
action plans, research reviews, and many others. We organized these resources into an Excel 
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document with columns listed by topic ( subject area), search string, producer organization, target 
audience, product name/title, product link, product type, producer organization link, product 
description, country / province of origin, and cost. 

Our analysis was an effort to reveal and map patterns from our search results. We met as a research 
team to share some of the trends we spotted initially through a review of the entire set of resources. 
For instance, we noticed that roughly 70% of the resources across subject areas were from U.S.-based 
organizations; that there was a large variety of producer organizations, many nonprofits; that there 
were more U.S.-based resources for special education and second language education than for the 
other subject areas; and that while the number of hits on each search string varied greatly, most 
search strings produced millions of hits, suggesting the overwhelming number of online-based 
resources available. (Even though a high proportion of search hits are duplicates or irrelevant, a 
search that generates a million hits would likely yield at least several thousand different and 
possibly relevant resources, which is far more than any practitioner would be able to explore.) 

Based on these initial observations, we inductively generated categories by which we could explore 
and compare the data on our indicators. These categories were developed also from prior team 
research (references suppressed for blind review) that noted the significance of such elements as type 
of organization and actors involved when examining KMb issues. The following categories were used 
to organize the typology (these three categories were tracked in the Excel document along with the 
other descriptive categories listed in italics above): 

Producer organization 

1. Higher education and their affiliated organizations 

2. Government/K-12 education 

3. Nonprofit (excluding 1 & 2) 

4. For-profit (excluding 1) 

5. Independent/individual (produced by individuals not affiliated with an organization) 

6. Other 

Targeted audience 

1. Teachers 

2. Administrators 

3. Other school practitioners 

4. Teachers and administrators 

5. More than one audience identified 

6. Other/uncertain 

Country of origin 

1. United States 

2. Canada 

3. International 

4. Online only/unknown 
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In order to run general descriptive statistics by subject area along these various dimensions 
(frequencies and percentages), we coded our data into these three nominal variables. Using SPSS V. 
10.0, we checked for inconsistencies in the data via frequency distributions and comparisons to our 
initial Excel spreadsheet. We then computed descriptive statistics comparing the frequencies of each 
variable as compared with the subject area (see Tables 1-3). 

Results 


Search Results 

Our search results confirm the enormous quantity and range of research-based resources educators 
will undoubtedly encounter online. Key results from our descriptive analysis follow. 

Producer Organization 

The selected products were produced by several kinds of organizations, from state- and district-level 
public institutions to private enterprises, foundations, think tanks, and nonprofit organizations. The 
highest percentages of resources were developed by nonprofits ranging from 33% (arts) to 70% 
(second language). All subject areas tended to have fewer resources produced by higher education 
organizations ranging from 6% (behavior management and literacy at 3 out of 50 resources for each) 
to 36% (the arts at 12 out of 33 resources). The highest percentage of for-profit resources originated 
in leadership at 28% (or 14 out of 50 resources). Very few resources were found in the “independent” 
category (a resource produced by an individual who is not affiliated with an organization, for 
instance, a blog or personal Web site). Higher percentages of resources were produced by the 
government/K-12 category at 26% (13 out of 50 resources) for behavior management and 22% for 
literacy/math (11 out of 50 resources; see Table 1). 


Table 1: Descriptive Statistics of Producer Organization by Subject Area 


Producer 

Organization 



Subject Area 




Special 

Education 

Behavior 

Management 

Literacy 

Second 

Language 

Leadership 

Math 

The 

Arts 

Higher 

7 

3 

3 

4 

5 

6 

12 

education 

14% 

6% 

6% 

8% 

10% 

12% 

36% 

Government/ 

5 

13 

11 

7 

9 

11 

5 

K-12 

10% 

26% 

22% 

14% 

18% 

22% 

15% 

Nonprofit 

31 

19 

28 

35 

19 

18 

11 


62% 

38% 

56% 

70% 

38% 

36% 

33% 

For-profit 

7 

12 

6 

3 

14 

10 

3 


14% 

24% 

12% 

6% 

28% 

20% 

9% 

Independent 

0 

0 

2 

1 

3 

3 

1 


0% 

0% 

4% 

2% 

6% 

6% 

3% 

Other 

0 

3 

0 

0 

0 

2 

1 


0% 

6% 

0% 

0% 

0% 

4% 

3% 

Total 

50 

50 

50 

50 

50 

50 

33 
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Variety of Resources 

The resources identified came in a wide variety of formats. Indeed, it is very difficult to construct a 
useful typology for the many different kinds available, so we chose not to include these data in our 
analysis. However, we observed the following as the most common types of resources across all 
subject areas: software programs, frameworks (model, strategy, action plan, approach, plan, system, 
outline, map, design), various forms of packaged material (handbooks, guidebooks, toolkits), Web 2.0 
resources (blog posts, online Microsoft PowerPoint, e-modules, online encyclopedias, discussion 
boards), and guidelines (tip sheets, bullet-point how-to, lesson plans, curriculum). 

Target Audience 

Target audience of the resources varied by subject area. A higher percentage of leadership products 
were targeted towards administrators (38%, or 19 out of 50 resources), while about half of the 
behavior management and arts resources targeted to teachers (48% and 46%, respectively). Several 
subject areas reported resources targeted to more than one audience, including second language 
(80%), literacy (72%), and mathematics (66%). All subject areas had at least some resources targeted 
to both teacher and administrators (see Table 2). 


Table 2: Descriptive Statistics of Targeted Audience by Subject Area 





Subject Area 




Targeted 

Special 

Behavior 


Second 



The 

Audience 

Education 

Management 

Literacy 

Language 

Leadership 

Math 

Arts 

Teachers 

17 

24 

5 

3 

1 

11 

15 

34% 

48% 

10% 

6% 

2% 

22% 

46% 

Administrators 

2 

2 

7 

2 

19 

2 

0 


4% 

4% 

14% 

4% 

38% 

4% 

0% 

Other school 








practitioners 

0 

0 

0 

0 

1 

0 

1 

0% 

0% 

0% 

0% 

2% 

0% 

3% 

Teachers and 

0 

o 

o 

jr 

0 

0 

A 

administrators 

A 

A 

0 

4 

0% 

4% 

4% 

10% 

0% 

0% 

12% 

More than one 
audience 

6 

8 

36 

40 

26 

33 

12 

identified 

12% 

16% 

72% 

80% 

52% 

66% 

36% 

Other 

25 

14 

0 

0 

3 

4 

1 


8% 

28% 

0% 

0% 

6% 

8% 

3% 

Total 

50 

50 

50 

50 

50 

50 

33 


Country of Origin 

Our search terms and therefore selected resources were in English only. For all subject areas with 
the exception of mathematics, 60-80% of the resources found online originated from the United 
States. This was particularly high for those resources found in behavior management (76%), second 
language (84%), and special education (74%), ranging from 37 to 42 out of 50 resources. There were 
very few resources reported for the “international” category (outside North America) in special 
education (4%), behavior management (4%), and second language (8%). Canadian-origin resources 
ranged around 10-20% for the various subject areas. Very rarely was it difficult to discern the 
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country of origin of a resource. The greatest percentage of international resources (outside of North 
America) came in mathematics (20%), the arts (18%), and literacy (16%). The resources that 
originated outside of North America were from several countries, including the United Kingdom, 
Singapore, The Netherlands, Australia, and New Zealand, with the majority from the United 
Kingdom and Australia (see Table 3). 


Table 3: Descriptive Statistics of Country of Origin by Subject Area 


Country 
of Origin 




Subject Area 




Special 

Education 

Behavior 

Management 

Second 

Literacy Language 

Leadership 

Math 

The 

Arts 

United 

37 

38 

32 

42 

33 

23 

22 

States 

74% 

76% 

64% 

84% 

66% 

46% 

67% 

Canada 

4 

6 

10 

4 

7 

8 

4 


8% 

12% 

20% 

8% 

14% 

16% 

12% 

Inter¬ 

2 

2 

8 

4 

6 

10 

6 

national 

4% 

4% 

16% 

8% 

12% 

20% 

18% 


7 

4 

0 

0 

4 

9 

1 

Unknown 

14% 

8% 

0% 

0% 

8% 

18% 

3% 

Total 

50 

50 

50 

50 

50 

50 

33 


Hits and Relevance 

We conducted an additional search in each of the seven areas using consistent search strings in 
order to compare number of hits and their relevance across the subject areas (in our original 
searches, team members had used search strings with different levels of specificity). We used the 
following search string format: “subject area”+education+resource+research based. Two pages were 
selected at random per topic area. We recorded the number of hits for the search string and counted 
the number of relevant hits for each of the two pages, as reported in Table 4. “Relevant” hits were 
those hits that linked directly to an education resource with a potential research basis in that subject 
area. There were 10 links per search page, so the total number of relevant resources was calculated 
out of 20. 
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Table 4: Search String Hits 


Topic Area 

Search String 

Results 

Random 

Pages 

# of Relevant* 
Resources 

Literacy 

Literacy+education+ 
resource+research based 

61,000,000 

2 & 7 

7/20 (35%) 

Math 

Math+education+ 
resource+research based 

229,000,000 

1 & 5 

12/20 (60%) 

Leadership 

Leadership+education+ 
resource+research based 

1,200,000,000 

3 & 8 

3/20 (15%) 

ESL/ELL 

ESL/ELL+education+ 
resource+research based 

6,590,000 

1 &4 

14/20 (70%) 

Special 

education 

Special education+ 
resource+research based 

957,000,000 

2 & 6 

11/20 (55%) 

Behavior 

management 

Behaviour management+ 
education+resource+ 
research based 

93,000,000 

3 & 4 

6/20 (30%) 

The arts 

The arts+education+ 
resource+research based 

1,280,000,000 

1 &9 

5/20 (25%) 


*Results reflect the number of hits found on the date and time of the search. 


As seen in Table 4, the least number of hits were found in ESL/ELL at 6.59 million, with the highest 
number in the arts and leadership, at 1.28 and 1.2 billion, respectively. Interestingly, it was the 
exact opposite with respect to research relevance. The percentage of relevant hits in ESL/ELL was 
70%, while the arts and leadership turned up 25% and 15% relevant hits respectively. 

Discussion 

Our data suggest the following conclusions. First, there is a huge amount of material available to 
educators on the Internet that claims to be research based. While we do not know just how many of 
these millions of hits are indeed relevant, it seems likely that even with large amounts of irrelevant 
hits or duplications there would still be many thousands of discrete resources available on each of 
these topics. Most educators would be overwhelmed by the volume of resources and information 
available online. 

However, the resources available appear to be limited in a number of ways. A very large proportion 
of such resources in English originate in the United States. Readers in other settings must take into 
account contextual differences that may render the findings or implications of American work less 
relevant to them. We also found that resources are produced by a range of organizations with 
varying motives—from promotion of their own organization to generating revenue. Of course such 
motivations can be consistent with producing high quality, objective materials but the potential user 
may want to take the originating organization into account, and in particular assess whether there is 
independent corroboration of any claims being made. One cannot automatically move from trusting 
the source to trust all messages from that source. 
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The abundance of materials poses two challenges. First, how is anyone to determine which resources 
warrant some fuller consideration? Given time constraints and a situation of an educator sifting 
through a plethora of online resources on the Web, one can imagine the challenge of making quick 
and accurate judgments on the quality, relevance, and usefulness of the resources. People are likely 
to confine themselves to the first few hits they find, or perhaps to give up altogether. However, 
because of the way search engines aggregate and rank search results, one cannot be confident that 
the first page or two of hits are indeed the most useful. If educators are more susceptible to resources 
presented in particular formats, it may be all too easy for them to select the wrong resources, such as 
those that falsely claim to be informed by research. The resource may have been swept up in the 
faddish idea of the time (Slavin, 1989), it may be marketed in formats that educators find more 
appealing (Cordingley, Bell, Evans, & Holdich, 2005), or it may have been recommended by peers 
based on anecdotal evidence (Mitton et al., 2007)—but the resource may have very little grounding in 
quality research. 

Our results also suggest the value of providing educators with additional guidance around searching 
and sorting through resources available online. Professional development around how to organize 
searches, how to filter results, and how to evaluate research and its quality may help educators in 
their efforts to quickly determine the reliability of resources claiming to be based on evidence, 
particularly when encountering a large number that are potentially relevant to informing their 
needs. This training might also include a component around helping educators become better 
informed and aware of their biases and obstacles when accessing and evaluating the resources 
(Cordingley, 2008; William & Coles, 2003). New developments in Web science and information 
systems may also provide direction in navigating and understanding the online space more generally 
(Hendler, Shadbolt, Hall, Berners-Lee, & Weitzner, 2008). 

However, the challenge is not primarily one for individual educators. Often, educators hear or learn 
about research ideas and resources via their informal networks or through administration (Daly, 
2011; Tseng, 2012); indeed, decisions about adopting programs or practices are made by school or 
system leaders rather than individual teachers. Therefore, while educators may benefit from 
training around how to search and identify research-based resources to use in their practice, it is 
even more important for school leaders and school systems to have effective processes for seeking, 
filtering, and sharing relevant resources. Several studies in KMb have shown that schools and 
systems typically have a weak capacity to do so (Levin, Cooper, Arjomond, & Thompson, 2010). 
Without formal processes for identifying and distributing quality research, school systems also run 
the risk of becoming receptive to certain ideas due to their own biases and search patterns and may 
adopt faddish or ineffective practices based on the support of one or a handful of people in the system 
(Levin, 2013). 

For research providers and intermediaries, which can also include school districts, these findings 
raise an additional challenge. It would seem rather wasteful for organizations to be producing yet 
more such resources—more guidebooks, tip sheets, or research summaries—when many thousands 
of them already seem to exist. Currently, a great deal of effort in many organizations is going into 
producing more resources; that effort might be better spent in reviewing existing materials and 
providing better access to high-quality existing materials for broader audiences. In other words, 
producers and intermediaries could shift their work from production to evaluation and broader 
sharing of existing resources they believe are of high quality. 
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The profusion of research-based resources found in our search raises several important questions 
and areas for future research. First, more can be done to understand the resources available to 
educators in the online environment, their research, claims to evidence, marketing efforts, and 
various formats of presentation. A further study could delve into the many thousands of hits to 
determine more carefully how many of them were actually relevant to a given search. Another data 
analysis would assess the degree to which resources are actually informed by research (e.g., by 
counting and examining citations). And yet another analysis could extend our very tentative typology 
of resources to get a better sense of what kinds of materials were available. These analyses could be 
combined to assess whether certain kinds of resources were more likely to be well grounded in 
research evidence than were others. In addition, given the complexity of the online environment, 
research can explore questions around how educators and school systems can better navigate this 
online space and use the Web more effectively to benefit their practice. And, given studies that show 
that few visitors accessed resources posted by various education organizations’ Web sites (Edelstein, 
Shah, & Levin, 2011), much more can be done to understand how these resources can be better 
shared in an online medium. 


Conclusion 

This study reports the results of a systematic search and descriptive analysis of research-based 
resources across seven subject areas available to educators in the online environment. The set of 
resources was presented in the form of a typology organized into several categories that included 
country of origin, target audience, and producer organization. The search results demonstrated an 
overwhelming number and large variety of resources, regardless of subject area, that were targeted 
to different audiences, produced by different kinds of organizations, and presented in a wide variety 
of formats, with notable patterns between subject areas. The study provides an initial step in 
understanding this vast online landscape and raised several important questions for future 
investigation in the under-researched area of technology uses in KMb. 
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